arXiv:1504.01782v3 [cs.NI] 25 Nov 2015 


N J I 


New Jersey’s Science & 
Technology University 



Profit Maximization for Geographical 
Dispersed Green Data Centers 


Abbas Kiani 
Nirwan Ansari 


TR-ANL-2015-002 

April 7, 2015 


Advanced Networking Laboratory 
Department of Electrical and Computer Engineering 
New Jersy Institute of Technology 


1 


Abstract 

This paper aims at maximizing the profit associated with running geographically dispersed green data 
centers, which offer multiple classes of service. To this end, we formulate an optimization framework 
which relies on the accuracy of the G/D/l queue in characterizing the workload distribution, and taps on 
the merits of the workload decomposition into green and brown workload served by green and brown 
energy resources. Moreover, we take into account of not only the Service Level Agreements (SLAs) 
between the data centers and clients but also different deregulated electricity markets of data centers 
located at different regions. We prove the convexity of our optimization problem and the performance 
of the proposed workload distribution strategy is evaluated via simulations. 
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I. Introduction 

The demand for online services including web search, online gaming, distributed file systems 
such as Google File System (GFS) (lj, and distributed Storage System such as BigTable (2j and 
MapReduce [3j is growing exponentially. This significant growing demand for online services 
has led to a multitude of challenges in Data Center Networks (DCNs) from DCN architecture 
design, congestion notification |4)-[12|, TCP Incast [13]-[1_5], virtual machine migration |16;|- 
0, to routing in DCNs [20). 

Most importantly, due to the gravity of preparing DCNs as a scalable and reliable computing 
infrastructure, online services run on hundreds of thousands of servers spread across large 
data centers have significantly craved electric power usage. Complying with such a growing 
demand in an environmentally friendly manner calls for innovations across different disciplines. 
Therefore, recently, studies on data centers have focused on reducing the energy consumption 
and accordingly the cost of electricity. These studies can be largely categorized into two main 
approaches: power management techniques and green data centers. The first approach, which 
investigates CPU and memory power consumption, aims at reducing the carbon footprints and 
the cost of electricity. The second approach, referred to as green data centers, not only tries to 
cut down the electricity consumption and its cost but also integrates renewable energy resources 
such as solar panels and wind farms into data centers, thereby promoting sustainability and 
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green energy. In the past few years, a small and cohesive body of work investigated workload 
distribution across multiple data centers and the researchers came up with a variety of policies 


and algorithms. The social impacts of geographical load balancing is explored in [231 and two 
distributed algorithms are provided that can be used to compute the optimal routing as well as 
provisioning decisions for Internet-scale systems. Another couple of research papers approach 
the problem by employing the mixed integer programming p2j, [23). Also, Ghamkhari et 


al. [241 addressed the trade-off between minimizing a green data center’s energy costs and 


maximizing its revenue. Also, Zhao et al. [25] took into consideration of dynamic VM pricing 
and designed a new algorithm to maximize the long-term cloud provider’s profit. Moreover, 


Kiani and Ansari [26 [ proposed a workload distribution strategy based on the notion of green 
workload and green service rate versus brown workload and brown service rate, respectively, 
and also real-time monitoring of the queue lengths. 

In this paper, we propose a new workload distribution strategy for geographically dispersed 
green data centers in which our strategy aims at maximizing the revenue and minimizing the 
energy expenditures. To this end, we formulate an optimization framework for profit maxi¬ 


mization which relies on the accuracy of the G/D/l queue [24], [27 [ in capturing the workload 
distribution. Moreover, our optimization-based workload distribution strategy taps on the merits 
of workload decomposition into green and brown workloads served by green and brown energy 
resources, respectively. In summary, we will address the following: 

• We develop a new model to maximize the profit of running geographically dispersed data 
centers. In our model, it is assumed that each data center is offering multiple classes of 
services and we take into account of individual SLA-deadline for each type of service. 
Also, we assume that each data center either has a renewable power source or is powered 
by a nearby wind or solar farm thereby taking into account of green energy. However, 
as the green energy resources may not be adequate to meet the QoS requirements for all 
incoming workloads, each data center is also provisioned by on-grid energy. Therefore, 
we further elaborate our model by taking into consideration of geographical electricity 
price diversity due to different electricity markets and time zones of the dispersed data 
centers. 

• Based on the developed model, we design an optimal workload distribution strategy in 
terms of the gained profit by the data centers. The profit is defined as revenue — cost by 
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considering the deadline, service income, penalty for the service requests of each class, 
and also both green and brown energy costs. Our strategy relies on the accuracy of the 
G/D/l queueing model in capturing the workload distribution. Furthermore, we prove the 
convexity of our optimization and therefore its appropriateness for practical purposes. In 


the optimization frameworks such as [24] which are proposed for a single data center, 
the service rate is the only decision variable. However, as our model is an extension for 
a group of data centers, our objective function and the constraints are functions of both 
allocated workloads to the data centers and the service rate at each data center. In other 
words, we maximize the profit by not only optimizing the service rates at data centers 
but also allocating optimized workload to each data center. To prove the convexity of 
our problem, we introduce the average number of dropped requests at each data center as 
an extended SLA constraint and based on that we can prove the convexity of the whole 
problem by using the convexity of the perspective of a function. 

• Our optimization model relies on the potential merit of the decomposition of the workload 
to the green and brown workload thereby taking into account of different costs and different 
environmental impacts of green and brown energy. In this way, we can allocate the green 
workload to the data centers based on the availability and cost variation of the green 
energy at different locations. However, for the brown workload, our strategy takes into 
account of electricity price diversity and hence distinguishes the data centers by the price 
of electricity. In fact, we take into consideration of not only the cost of brown energy but 
also one time capital and maintenance expenses of renewable energy. Therefore, unlike 
some of the existing works in the literature, our optimal profit is not under the assumption 
that local renewable generation is always less than the local power consumption. 

• We evaluate the proposed workload distribution strategy via simulations and demonstrate 
that it outperforms the existing workload distribution strategies in terms of the total profit. 

The rest of paper is organized as follows. Sections [TT] and III describe the system model 


and problem formulation. In Sections IV we propose our optimization framework. Finally, 
Sections [V] and VI present numerical results and conclude the paper, respectively. 
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Figure 1. System Model. 


II. System Model 

Figure [I] shows the proposed system model in which we consider a group of \N\ data centers 
dispersed at different regions. Each data center is equipped with a collection of M, homogeneous 
servers. 

The data centers are supplied by multiple types of power. The major power supply of each data 
center is on-grid or brown energy. The data center has to pay brown energy prices according 
to its contract with the power company. The electricity pricing contract for each data center 
depends on the electricity markets at the data center’s location. If the market is regulated, the 
electricity price has a flat rate during the day. On the other hand, if the region is following a 
deregulated market, the price of electricity is varying. In most cases, the data center pays less 
during off-peak hours and more during on-peak period. Therefore, we note the price variability 
among data centers located at different locations and time zones. 

To reduce the cost of electricity and to capitalize on the environmental and sustainability 
advantages of green energy, we assume that each data center either is equipped with a renewable 
power source or has access to a nearby renewable energy source such as solar panels or a wind 
farm. It is worth mentioning that we assume the available renewable energy at each data center 
can only be used to supply power locally. 

Each data center is offering \J\ multiple classes of service like web services, video streaming, 
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etc. Each type of service has its specific deadline according to the SLA. 

The service requests are initiated by users and arrive at the workload distribution center. One 


or a group of servers can serve as the workload distribution center [28]. These servers can be 
treated as the front-end devices that exist in multi-data center Internet services like Google and 


Itunes [29]. The distribution center facilitates workload flexibility at the demand side. In other 
words, this center inspects the arriving requests from all users and manages the distribution of 
the incoming workload to the geographically dispersed data centers based on the availability of 
green energy and the price of electricity. In our formulation, the total power consumption at 


each data center takes into account of the Base Load and Proportional Load [24], 

Total Power Consumption at data center i = 

TTlj \ Pj.dle T (^usage 1 ' jEpeak ] T m^ [(Ppeak Pidle)Uj\i (1) 


where the base load, mi[Pidi e + (E usage — E)P P eak\, indicates the power consumption even when all 
of the turned on servers are idle. The proportional load, m.i{(P peak — P id i e )Ui\, is the extra power 
consumption which is proportional to the CPU utilization of the servers, Ui, and accordingly 
to the workload. It is worth mentioning that both base and proportional loads are computed 
based on the number of switched on servers, idle power, P ld i e , an d average peak power of 
a single server, P pea k ■ Moreover, due to different energy efficiencies at different data centers, 
the definition of the total power consumption incorporates the Power Usage Effectiveness (PUE) 
ratio, E usage , thereby amalgamating the power consumption at facility for cooling, lighting, and 
other overhead. [ 30] ]. 


III. Problem Lormulation 

We divide the running time of the data centers into a sequence of time slots at equal length, 
T, e.g., a few minutes. Our goal is to maximize the data centers’ total profit during the interval 
T. To this end, we propose an optimization problem to be solved at the beginning of each time 
slot in which we update the number of turned on servers as well as the allocated workload to 
each data center. Note that for the analysis, we consider a single time slot, e.g., A as the time 
slot of interest, and omit the explicit time dependence in the notations. 

At the beginning of each time slot, we allocate the workload (total number of service requests) 
to the data centers based on the availability of green energy and the price of electricity. As the 
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renewable energy and brown energy incur different costs and different environmental impacts, 
we decompose the total workload into the green and brown workload. In fact, we distinguish 
the servers at each data center based on the energy which is utilized to power them. Some 
of the servers are turned on and powered by the available green energy (green servers), and 
the others, if needed, by purchasing brown energy (brown servers). Therefore, the distinction 
between green and brown workloads is made mainly based on the server which is utilized to 
serve the workload. Specifically, the requests served by a green server are defined as the green 
workload and similarly those by a brown server the brown workload. 

The data center’s profit is modeled as Revenue — Cost, where the data center’s revenue is 
calculated based on the QoS requirements satisfaction and the cost indicates the energy cost. 
Owing to the limited computational resources at the data centers, the allocated requests to a 
data center are first placed in a queue before they can be processed by any available server. 
Accordingly, to satisfy the QoS requirements, the queueing delay for each service request should 
be limited by a deadline. If the data center can handle the service requests by the deadline, it 
receives the service income. Otherwise, it has to pay penalty to its customers. These three 
parameters, i.e., the deadline, service income, and penalty, depend on the type of service and 


are determined by the SLA [24], [311. Thus, we assume that the waiting requests of different 
classes of service at each data center are placed in different queues. Denote Dj, Sj, and 
as the deadline, service income, and penalty for the service requests of class j, respectively. 


The service requests that are not handled by the deadlines are discarded 321. In our problem 
formulation which is based on the workload decomposition, we distinguish the profit gained by 
serving green workload from the brown workload as the green and brown profit, respectively. 
To this end, we assume the green and brown requests of each class are placed in two different 
queues at a data center. In the next two subsections, we will formulate the green and brown 
profits. 


A. Green Profit Formulation 

We assume that the request rate of each class of service at the workload distribution center is 
a random process with an arbitrary and general probability distribution function, and \ n . denotes 
the service request rate of class j at time n. Let \ 3 be the average rate of receiving service 
requests of class j at the workload distribution center within time slot A of length T. Also, 
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cr| denotes the variance of the class j service request rate’s probability distribution function. 
Request interarrival times are assumed to be much shorter than a time slot duration, so that the 
request allocation can be based on the average arrival rate during the time slot. 

We allocate fraction of the service requests to the data center i’s green servers. These 
requests are first placed in a particular queue on green servers. The input process to this queue, 
i.e., A 9n .. = nj, has the same general probability distribution function as the request rate of 
class j. Therefore, X gij =A 0 and a 2 .. = (-y^) 2 cr| are the mean and variance of the input process 
to the corresponding queue, respectively. 

Based on the aforementioned QoS model, the green revenue earned by the data center i for 
serving the green requests of different classes of service within a time slot can be calculated as, 

•/'•/,,,) = E^iQ 1 - PdKi, VgaWiKjT ~ P l{Kv VgahiKi 1 ')’ where P dK r tha) 
denotes the probability that the waiting time for a service request of class j exceeds its SLA- 
deadline. Note that fi g denotes the green service rate, i.e., the rate that the requests of class j 
are removed (i.e., served) from the corresponding queue by the data center V s green servers. 

To obtain P L (X gij , the SLA-deadline is translated into the loss probability of a G/D/l 
queue. In a nutshell, it is assumed the service rate that the service requests are removed from 
the queue, i.e., n 9ij , is fixed over the time slot. Thus, for instance, if there are Q ig number of 
requests waiting in the queue upon the arrival of a new service request, it takes seconds 


t 1 9i 


until the new request can be handled by any available server. If < D~, then the new request 


I 1 Si 


can be handled before the deadline. Therefore, the SLA-deadline can be modeled by a finite-size 
queue with length n gij Dj. In other words, in order to handle a new request by the SLA-deadline, 


it has to enter a queue with length ji g .. D 3 [24], According to queueing analysis [27 ], the loss 


probability of the finite-size queue can be accurately estimated from the tail of the queue length 
distribution for any general probability distribution. However, it is known that the estimation 
yields the highest level of accuracy when the service request rate is characterized by a Gaussian 
process (27J. Therefore, through out the rest of this paper, the request rate of each class of 
service, accordingly the input process to the queues is assumed to be a Gaussian process, and 
the loss probability can be obtained as, 


p L(X gij} fi gij ) oi{\ gij , fi gij )e 




( 2 ) 
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a (^gij > t^gij) 

1 


3 ij~ X 3 ij) 

9 /t 2 


-(r-Ag .,) 2 


X gij y/2ira, 


d9i. 


dr , 


9ij 




and for each n > 1, 


j ^gij) — 


((Dj di)n gij + n((i gi . A gij )) 


(3) 


(4) 


nC' A9ij (0) + 2E;r i 1 (n-/)C' A9i3 (/)’ 

where C A (l) is the autocovariance of the class j service request rate’s probability function 
at data center i, and we have cr? = C\ ..(0). Also, di is the network delay experienced by a 
request from the workload distribution center to data center i. 

The green power consumption at each data center depends on the number of switched on green 
servers as well as the CPU utilization of each green server. The total number of switched on green 
servers at data center i can be expressed based on the total green service rate as m gi = V^ 
where each server can handle k 3 service requests of class j per second. Also, within the interval 


of T, each switched on green server handles 




requests of class j [241. Thus, 


the total CPU busy time of each server can be obtained as Y^j= i T(1 ^ gy dividing 


the total server busy time by T, we have the CPU utilization U, h = ^'/h ^ ^ . 
Therefore, referring to the definition of power consumption in (jTj), the total green power con¬ 
sumption in data center i at the time of interest can be expressed as, 


\J I 


Ei(X gij , H 9ij ) — (yPidle + ( E usage ~ 1 )Ppeak) + 


3 =1 


\J\ 


(Ppeak Pidle ) 

3 =1 


(1 PL(X gij , ld gij ))X gij 


kn 


(5) 


Note that the total number of the green servers at each data center, and accordingly the green 
service rate is limited by the available green energy at the time slot of interest. Let W % be the 
available green energy at data center i within the time slot. Wi is predicted at the beginning of 
the time slot, and depends, for example, on wind speed and solar irradiance. Similar to some 


other published papers in the literature such as [24], 28 ] it is assumed that the time slot is small 
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enough (e.g., every few minutes). Therefore, while the amount of renewable energy is changing 
at different time of a day, it is reasonable that solar irradiance and wind speed are relatively stable 
within a slot. We assume C 9i is the cost of renewable energy at data center i. The cost of green 
energy generation includes one time capital and maintenance expenses. The average unit cost of 
renewable energy can be obtained by averaging over the total amount of energy generated during 
the whole operation period. Therefore, the total green profit gained by all the data centers during 
the time slot of interest can be calculated as Profit g = ^1=1 (/T, (A ry . ty , p g ..)-C g TEfiX 9ij , p 9i .)). 


B. Brown profit formulation 

If green energy generation is not adequate to serve all incoming workload, brown energy 
is purchased. Brown energy is considered as an additional resource to power on additional 
servers referred to as the brown servers. We allocate X bn = --^-A nj service requests, as the 
brown requests, to the data center V s brown servers. These requests are first placed in their 
particular queue on brown servers, and X bij 0 and o%.. = arc the mean and variance 

of the input process to the queue, respectively. When using brown energy, we note the different 
deregulated electricity markets of data centers located at different regions. Denote C bi as the price 
of electricity at data center i within the time slot of interest. In order to benefit from the electricity 
price diversity, the distribution center can employ the day-ahead electricity price forecasting 


methods [33], [34]. Therefore, the total brown profit gained by all the data centers during the 
time slot of interest can be calculated as, Profit b = p bij ) - C bi TE bi (X bi ., p bij )). 

In the next section, we propose an optimization framework for the service request distribution. 
The objective of our framework is to maximize the total profit earned by the data centers within 
each time slot. Our optimization framework uses the results of renewable energy and electricity 
price forecasting methods. 


IV. Optimization Framework 

In order to maximize the total profit earned by the data centers, we update the allocated 
workload and the service rates for each data center. In fact, we seek to maximize the total profit 
by optimizing the allocated green and brown requests (i.e., X fHj and X bij ) as well as the green 
and brown service rates (i.e., p gij and p bij ) within each time slot. To this end, the following 
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optimization problem is proposed to be solved at the beginning of the time slot of interest, 


maximize ( Profit g + Profitf) 

Hj *^9ij ij 

(6) 

subject to 


0 < x 9ij < H 9ij , Vi e N , Vj g J, 

(7) 

0 < A bi:j < Hb i:j , Vi G N, Vj G J, 

(8) 

iv, 

j —1 'a? 1 peak-^usage 

(9) 

1 N\ 


V+Ki + 'V;.'/) = Vj G J, 

(10) 

2=1 


^gijPL{X 9ij , fi 9ij ) < THj : Vi G N, Vj G J, 

(ID 

A/ i( ;/V(A/, ;r //y ; ) < THj, Vi G N, Vj G J, 

(12) 


where the inequality constraints (j7]), ([8]) are to lower bound the service rate of each queue by the 
average of the input process to that queue and are necessary for stabilizing the service request 
queue. In addition, the inequality constraint (|9]) is used to limit the green service rates by the 
available renewable energy in which we make full CPU utilization assumption. Also, we use 
equality constraint ( |T0| ) to allot all the requests of each class to the data centers based on the 
average rate of receiving service requests. Moreover, by inequality constraints ([TT}, < [I2| ), we 
add an extended SLA requirement in which the average number of dropped requests at each 
queue is upper bounded by a constant THj. 

The proposed optimization problem is a convex optimization problem, as proven in the 
following theorem, and consequently can be solved by efficient optimization techniques, such 
as the interior point method (IPM). 
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Theorem 1 The constrained optimization problem ([6]) is a convex optimization problem if data 
centers are profitable for each class of service and 

P 9t:i > 1 and p bij > 1,V i,j (13) 




Figure 2. Wind power generation. 


Figure 3. Price of electricity. 




Figure 4. The total incoming workload. 


Figure 5. Normalized profit gain. 


The proof of Theorem [I] is given in the appendix. It is worth mentioning that the G/D/l model 
in [27] is valid only for the range of service rates, p 9ij > X gij and p b > X g .., which we have 
already considered in our constraints. Therefore, even if we do not allocate any workload to a 
data center, the service rate has to be set greater than one for the problem to be convex. 
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V. Simulation Results 

We consider \N\ = 3 data centers offering \J\ = 2 different classes of service. Each data 
center is integrated with a wind farm as a renewable power source. It is assumed that the data 
centers are located at three different regions with deregulated electricity market. Our simulation 
data are based on the trends of wind power and electricity price shown in Figures [2] and [3] 
respectively, which are updated every hour. We simulated the total workload of two classes of 
service using two sample days of the requests made to the 1998 World Cup web site [35] shown 
in Figure [4] Also, for each turned on server, we have assumed P pen k = 0.2 kw, P id i e = 0.1 kw, 
and E usa g e 1.2. 


(a) 



m 



♦-Proposed design 

♦-Design in [24] adapted for the case of multiple data centers 


' 0 10 20 30 40 50 60 70 80 90 100 

Relative increase in wind power at all data centers (percent) 


Figure 6. Performance comparison between the profit gain of the proposed design and design in (24) adopted for the case of 
multiple data centers, (a) 24 hours operation, (b) One time slot. 


Figure [5] compares the normalized profit gained by running three data centers. As shown in 
this figure, the curves represent the normalized profit of our proposed optimization problem and 


the design which is based on M/M/1 queueing [36J. The normalized profit gain is calculated 

as (Profit - Pro fit Base ) / (Prof it Max ~ Profit BaS e) where Profit Base is the profit obtained 


when /i = A and Pro fit Max is the maximum of the profit curve obtained by simulation [241. 
We can see that the proposed design outperforms the normalized profit gain of M/M/1 queueing 
because the G/D/l queueing model can capture the workload distribution more accurately than 
M/M/1. Also, Figure [6] demonstrates the better performance of our proposed design than the 
design in j24j adapted for the case of multiple data centers. While Figure [6} a) compares the 
gained profits of 24 hours operation of the data centers, Figure |6][b) shows the gained profit of a 
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Figure 7. Allocated green workload to the data centers, (a) First class of service, (bj Second class of service. 
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Figure 8. Allocated brown workload to the data centers, (a) First class of service, (b) Second class of service. 


sample time slot versus the relative increase in green energy. Figure [TJ a) and (b) demonstrate the 
allocated green workloads of the first and second class of service to each data center, respectively. 
For example, the trend of wind power indicates that after hour 15 most of the green workload 
is assigned to data center 1 where the highest wind power is available. However, from hours 10 
to 13, the available wind power at data center 1 is lower than the other data centers, and thus 
less of the green workload is allocated to this data center. Finally, Figure [8] shows the allocated 
brown workloads of the first and second class of service to each data center. For example, as 
shown in the Figure [8ja), from hours 8 to 11, all of the left over of the requests of both classes 
(the requests that are not served by green energy) are allocated to data center 2 where the price 


November 26, 2015 


DRAFT 

















































14 


of electricity is the lowest. 


VI. Conclusion 

In this paper, we have developed a new model to maximize the profit of running geographically 
dispersed data centers. Our model considers multiple classes of service and takes into account 
of individual SLA-deadline for each type of service. Furthermore, our model is elaborated by 
taking into consideration of geographical electricity price diversity due to different electricity 
markets at each data center’s location and the availability of renewable energy. 

Based on the developed model, we have designed an optimization-based workload distribution 
scheme that relies on the accuracy of G/D/l queue in characterizing the workload distribution 
and the workload decomposition to the green and brown workload. We have also proven the 
convexity of the formulated optimization problem and evaluated the performance of our workload 
distribution scheme via extensive simulations. 

References 

[1] S. Ghemawat, H. Gobioff, and S. T. Leung, “The google file system,” ACM SIGOPS Operating Systems Review, vol. 37, 
no. 5, pp. 29-43, 2003. 

[2] F. Chang, J. Dean, Hsieh W. C. Ghemawat, S., D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, 
“Bigtable: A distributed storage system for structured data,” ACM Transactions on Computer Systems, vol. 26, no. 2, pp. 
1-26, 2008. 

[3] J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing on large clusters,” Communications of the ACM, vol. 
51, no. 1, pp. 107-113, 2008. 

[4] Y. Zhang and N. Ansari, “On architecture design, congestion notification, tcp incast and power consumption in data 
centers,” IEEE Communications Surveys & Tutorials, vol. 15, no. 1, pp. 39-64. 2013. 

[5] D. Bergamasco and R. Pan, “Backward congestion notification version 2.0,” IEEE 802.1 Meeting, September 2005. 

[6] D. Bergamasco, “Ethernet congestion manager,” IEEE 802.1Qau Meeting, March 2007. 

[7] J. Jiang, R. Jain, and C. So-In, "An explicit rate control framework for lossless etherent operation,” in IEEE International 
Conference on Communications (ICC). Beijing, China, May 2008, pp. 5914-5918. 

[8] C. So-In. R. Jain, and J. Jiang, “Enhanced forward explicit congestion notification (e-fecn) scheme for datacenter ethernet 
networks,” in Symposium on Performance Evaluation of Computer and Telecommunication Systems. Edinburgh, UK, Jun 
2008, pp. 542-546. 

[9] M. Alizadeh, B. Atikoglu, A. Kabbani, A. Lakshmikantha, Prabhakar B. Pan, R., and M. Seaman, "Data center transport 
mechanisms: Congestion control theory and ieee standardization,” in 46th Annual Allerton Conference on Communication, 
Control, and Computing. Allerton House, UIUC, Illinois, Sep 2008, pp. 1270-1277. 


November 26, 2015 


DRAFT 



15 


[10] M. Alizadeh, B. Atikoglu, A. Kabbani, A. Lakshmikantha, Prabhakar B. Pan, R., and M. Seaman, “Data center transport 
mechanisms: Congestion control theory and ieee standardization,” in 46th Annual Allerton Conference on Communication, 
Control, and Computing. Allerton House, UIUC, Illinois, Sep 2008, pp. 1270-1277. 

[11] Y. Zhang and N. Ansari, “On mitigating tcp incast in data center networks,” in IEEE International Conference on 
Computer Communications. Shanghai, China, April 2011. 

[12] Y. Zhang and N. Ansari, “Fair quantized congestion notification in data center networks,” IEEE Transactions on 
Communications , vol. 61. no. 11, pp. 4690-4699, November 2013. 

[13] A. Phanishayee, E. Krevat, V. Vasudevan, D. G. Andersen, G. R. Ganger, Gibson G. A., and S. Seshan, “Measurement 
and analysis of tcp throughput collapse in cluster-based storage systems,” in USENIX Conference on File and Storage 
Technologies. San Jose, USA, Febuary 2008, pp. 1-14. 

[14] Y. Chen, R. Griffith, J. Liu, R. H. Katz, and A. D. Joseph, “Understanding tcp incast throughput collapse in datacenter 
networks,” in ACM workshop on Research on Enterprise Networking. Barcelona, Spain, August 2009, pp. 73-82. 

[15] V. Vasudevan, A. Phanishayee, H. Shah, E. Krevat. D. G. Andersen, G. R. Ganger, G. A. Gibson, and B. Muller, “Safe and 
effective finegrained tcp retransmissions for datacenter communication,” in ACM SIGCOMM. Barcelona, Spain, August 
2009, pp. 303-314. 

[16] F. Machida. D. S. Kim, J. S. Park, and K. Trivedi, "Toward optimal virtual machine placement and rejuvenation scheduling 
in a virtualized data center,” in IEEE International Conference on Software Reliability Engineering Workshops , November 
2008, pp. 1-3. 

[17] F. Machida and A. Sahoo, “On theory of vm placement: Anomalies in existing methodologies and their mitigation using 
a novel vector based approach,” in IEEE International Conference on Cloud Computing, July 2011, pp. 275-282. 

[18] H. Liu, H. Jin. X. Liao, C. Yu, and C. Xu, “Live virtual machine migration via asynchronous replication and state 
synchronization,” IEEE Transactions on Parallel Distributed Systems, vol. 12, no. 22, pp. 1986 - 1999, March 2011. 

[19] V. Shrivastava, P. Zerfos, K. Won Lee, H. Jamjoom, Y. H. Liu, and S. Banerjee, “Application-aware virtual machine 
migration in data centers,” in IEEE International Conference on Computer Communications. Shanghai, China, April 2011, 
pp. 66-70. 

[20] K. Chen, C. Hu. K. Zhang, K. Zheng, Y. Chen, and A. Vasilakos, “Survey on routing in data centers: insights and future 
directions,” IEEE Network, vol. 25. no. 4. pp. 6-10. July-August 2011. 

[21] Z. Liu, M. Lin, A. Wierman, S. Low, and L. Andrew. “Greening geographical load balancing,” in ACM SIGMETRICS 
joint international conference on Measurement and modeling of computer systems, 2011, pp. 233-244. 

[22] L. Rao, X. Liu, L. Xie, and W. Liu, “Minimizing electricity cost: optimization of distributed internet data centers in a 
multi-electricity-market environment,” in IEEE Conference on Computer Communications (INFOCOM), 2010, pp. 1-9. 

[23] J. Li, Z. Li, K. Ren, and X. Liu, “Towards optimal electric demand management for internet data centers,” IEEE 
Transactions on Smart Grid, vol. 3, no. 1, pp. 183-192, 2012. 

[24] M. Ghamkhari and H. Mohsenian-Rad, “Energy and performance management of green data centers: A profit maximization 
approach,” IEEE Transactions on Smart Grid, vol. 4, no. 2, pp. 1017-1025, 2013. 

[25] J. Zhao, H. Li, C. Wu, Z. Li, Z. Zhang, and F. Lau, “Dynamic pricing and profit maximization for clouds with geo- 
distributed datacenters,” IEEE Infocom, 2014. 


November 26, 2015 


DRAFT 



16 


[26] A. Kiani and N. Ansari, “Towards low-cost workload distribution for integrated green data centers,” IEEE Communications 
Letters, vol. 19, no. 1, pp. 26-29, 2015. 

[27] H. S. Kim and N. B. Shroff, “Loss probability calculations and asymptotic analysis for finite buffer multiplexers,” 
IEEE/ACM Transactions on Networking, vol. 9, no. 6, pp. 755-768, December 2011. 

[28] M. Ghamkhari and H. Mohsenian-Rad, “Optimal integration of renewable energy resources in data centers with behind- 
the-meter renewable generator,” in IEEE International Conference on Communications (ICC), 2012, pp. 3340-3344. 

[29] K. Le, R. Bianchini, M. Martonosi, and T. D. Nguyen, “Cost-and energy-aware load distribution across data centers,” in 
Workshop on Power Aware Computing and Systems. Big Sky, MT, October, 2009. 

[30] United States Environmental Protection Agency, “Epa report on server and data center energy efficiency,” Final Report 
to Congress, Aug. 2007. 

[31] D. Kusic, J. O. Kephart, J. E. Hanson, N. Kandasamy, and G. Jiang, “Power and performance management of virtualized 
computing environments via lookahead control,” Cluster computing, vol. 12, no. 1, pp. 1-15, 2009. 

[32] C. Wilson, H. Ballani, T. Karagiannis, and A. Rowtron, “Better never than late: Meeting deadlines in datacenter networks,” 
in ACM SIGCOMM Computer Communication Review. ACM, 2011, vol. 41, pp. 50-61. 

[33] L. Wu and M. Shahidehpour, “A hybrid model for day-ahead price forecasting,” IEEE Transactions on Power Systems, 
vol. 25, no. 3, pp. 1519-1530, 2010. 

[34] P. Areekul, T. Senjyu, H. Toyama, and A. Yona, “A hybrid arima and neural network model for short-term price forecasting 
in deregulated market,” IEEE Transactions on Power Systems, vol. 25, no. 1, pp. 524-530, 2010. 

[35] http://ita.ee.lbl.gov/html/contribAVorldCup.html. 

[36] L. Rao, X. Liu, L. Xie, and W. Liu, “Coordinated energy cost management of distributed internet data centers in smart 
grid,” IEEE Transactions on Smart Grid, vol. 3, no. 1, pp. 50-58, 2012. 

[37] S. Boyd and L. Vandenberghe, Convex optimization, Cambridge university press, 2009. 

[38] M. Abramowitz and I. Stegun, Handbook of mathematical functions: with formulas, graphs, and mathematical tables, 
2007. 


Appendix 

Proof of Theorem [TJ To show the convexity of the proposed optimization problem, we 
require to prove |37|: 

• The objective function, i.e., Prof it 9 + Profits is concave. 

• The inequality constraint functions are convex. 

• The equality constraint functions, i.e., ^|=[(A 9i . + A^.) — A j, are affine. 

Since the corresponding functions of the constraints ([7]), ([8]), ([9]) and ( [TO] ) are all linear, we start 
by proving the convexity of the following function, 

/(•VftJ = K,PAKpN v )-THj, Vi 6 in Vi € |J|. (14) 
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From (|2]), as e x is non-increasing, we have 


I 1 a.,)* * Ma{Xa v’ liB v ) . 


(15) 


Since max preserves convexity [37] and TIl } is constant, the function /(A Si .,// Si .) is proven to 
be convex if we can prove the following function, 

/»('W= KM x 9ii^9ij) e ~^ Mri{Xaii ^ 9ij \ (16) 

is convex for each n > 1. 

After reordering the terms in ([3]), we can show that, 

a (X gij , Pgij) — 


_ _ \ \ (A * g jj Xa ij-~ r oo 

a 9ij ^ \Nij A 9ij) „ / 




9ij 

,A, 


9ij 


9ij 


e 2 du] 


(17) 


By substituting a 9ij = ( )a 3 in (17) and (4) respectively, and after simple algebraic manipu¬ 
lation we have, 

a (^9ij 1 P'Qij) 


a 


1 / P9ij 


11 Cv/a," -1 " 


and 


2 C2 '•Ag.. 1 


((D j -d i + n)(^-l) + (D j ^d i )f 


e 2 <7u] 


C v . v A„. . -O 


d’gij) — 


Pn 


where C Vj = ^ is the coefficient of variation of the class j 9 s service request rate. Also, 


n—l 


Pn 


n^vj + 2 y: (w—z) 


^A,(Z) 


1=1 




(18) 


(19) 


( 20 ) 


Equations (18) and (19) indicate that f n (X g , n is the perspective of the following function, 

dn(d'gij') = cx(n gij )e^ M ^\ (21) 

where 

a {dgij) = 

( 22 ) 


a (dgij) 

a,. 1 


[! “ “ !)e / e 2 du] 




C Vj (^gij 1) 


and 


MniHgii) — 


((Dj — dj + n )(dgij - 1 ) + {Dj - dj)Y 
Pnj 


(23) 
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If g n (/j,g..) is convex, so is its perspective function f n (X 9ij ,n 9ij ) [37]. Therefore, w 
our proof by proving the convexity of g n (p gij ). Let’s define 


, a iVgij i) 

a. 


Then, we have 


9n(t) = a(t)e 3 M « (t) 

C 2 - 2 

a(t) = —/==[! — J e 


and 


M n (t) = 


((Dj — di + n)C Vj t + (Dj — 
Pn-j 


Then, the function g n (p g , ) is proven to be convex if we can show for each n > 1, 


9n(t) = e 2 M " (t) (a"(f) +a(t) 




M"(t i 


By simple algebra, we can show that, 


'U\ ^ k 

“ W = - AS 


and 


a"(t ) = (t 2 + 3)a(£) — 




By substituting (29) and (30) in g”(t), we have 


+(3 + - Mm )t _ A<(i) + - ()] 


Now, we show (28) for all t > 0. 


First, since nC 2 . < p n . < n 2 C 2 ., we can show that, 

> t z - 1 


K (t) M"(t) ^ . 


Then, from the following upper and lower bounds [381 

2 


._= < e 2 e 2 du < — 

t + Vt 2 + 4 Jt t+\/f 2 +- 


we have, 




[1-^=] < c(t) < ■%=[! _ 


2t 


f+J * 2 + ! 


y/2% t + \/f 2 + 4 
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which indicates a(t) >0 and we can show that 


c„ 


\^2na(t) 


> ( t +' /FT V > e- + i 


(35) 


From (35) and (32), the following inequality holds, 

a(t)e~ 


m > 


[f 3 — t 2 M' n (t) 


(36) 


+ (3 + f - 1 )t - M' n (t) + {f + 1 ){M' n {t) - t)} 

= a{t)e-s MnW {t 2 + 1 ) >0 

Therefore, for all t > 0, i.e., /i fhj > 1, g n (fx gij ) and consequently f(X 9ij , ix gij ) is convex. The 
convexity of the following function: 


HK’iO = KWK-iO ~ TH ’< Vi ew,y#e T 


(37) 


can be similarly be proven and we conclude the convexity of inequality constraints <HQ>. 

Now, we prove the concavity of the objective function. Note that the nonnegative weighted sum 


of concave functions is concave [37]. Also, the functions — X gij P L (X gij , jJL gij ) and —X bi .P L (X b ..,n bi .) 
are concave. Therefore, by rewriting the objective functions based on *-X gij PL,{X gij , fj, 9ij ) and 
-X biJ P L (X bii ,ix bii ), we can show that if the data centers are profitable for each class of service, 


i.e., 


+ 7? 


P'penk Pi 


idle 


kj 


ma x(C bi ,C, 


9i) 


> Sj - P '™“. — max(C t „ C 9 .) > 0, V * 

Kj 


(38) 


the objective function is concave and the proof is complete. 
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